﻿@{
    Layout = "~/Views/Shared/_Layout_Turorials.cshtml";
}
<h1>Orleans Streams </h1>

<p>Orleans v.1.0.0 added support for streaming extensions to the programing model. Streaming extensions provide a set of abstractions and APIs that make thinking about and working with streams simpler and more robust. Streaming extensions allow developers to write reactive applications that operate on a sequence of events in a structured way. The extensibility model of stream providers makes the programming model compatible with and portable across a wide range of existing queuing technologies, such as <a href="http://azure.microsoft.com/en-us/services/event-hubs/">Event Hubs</a>, <a href="http://azure.microsoft.com/en-us/services/service-bus/">ServiceBus</a>, <a href="http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-queues/">Azure Queues</a>, and <a href="http://kafka.apache.org/">Apache Kafka</a>. There is no need to write special code or run dedicated processes to interact with such queues.</p>

<h2 id="why-should-i-care">Why should I care?</h2>

<p>If you already know all about <a href="http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/">Stream Processing</a> and are familiar with technologies like <a href="http://azure.microsoft.com/en-us/services/event-hubs/">Event Hubs</a>, <a href="http://kafka.apache.org/">Kafka</a>, <a href="http://azure.microsoft.com/en-us/services/stream-analytics/">Azure Stream Analytics</a>, <a href="https://storm.apache.org/">Apache Storm</a>, <a href="https://spark.apache.org/streaming/">Apache Spark Streaming</a>, and <a href="https://msdn.microsoft.com/en-us/data/gg577609.aspx">Reactive Extensions (Rx) in .NET</a>, you may be asking why should you care. <strong>Why do we need yet another Stream Processing System and how Actors are related to Streams?</strong> <a href="Streams-Why">“Why Orleans Streams?”</a> is meant to answer that question.</p>

<h2 id="programming-model">Programming Model</h2>

<p>There is a number of principles behind Orleans Streams Programming Model.</p>

<ol>
    <li>Following the philosophy of <a href="https://github.com/dotnet/orleans/wiki/Grains">Orleans virtual actors</a>, Orleans streams are <em>virtual</em>. That is, a stream always exists. It is not explicitly created or destroyed, and it can never fail.</li>
    <li>Streams are <em>identified by</em> stream IDs, which are just <em>logical names</em> comprised of GUIDs and strings.</li>
    <li>Orleans Streams allow to <em>decouple generation of data from its processing both in time and space</em>. That means that stream producer and stream consumer may be on different servers, in different times and will withstand failures.</li>
    <li>Orleans streams are <em>lightweight and dynamic</em>. Orleans Streaming Runtime is designed to handle a large number of streams that come and go at a high rate.</li>
    <li>Orleans stream <em>bindings are dynamic</em>. Orleans Streaming Runtime is designed to handle cases where grains connect to and disconnect from streams at a high rate.</li>
    <li>Orleans Streaming Runtime <em>transparently manages the lifecycle of stream consumption</em>. After an application subscribes to a stream, from then on it will receive the stream’s events, even in presence of failures.</li>
    <li>Orleans streams <em>work uniformly across grains and Orleans clients</em>.</li>
</ol>

<h2 id="programming-apis">Programming APIs</h2>

<p>
    Applications interact with streams via APIs that are very similar to the well known <a href="https://msdn.microsoft.com/en-us/data/gg577609.aspx">Reactive Extensions (Rx) in .NET</a>, by using <a href="https://github.com/dotnet/orleans/blob/master/src/Orleans/Streams/Core/IAsyncStream.cs"><code class="highlighter-rouge">Orleans.Streams.IAsyncStream&lt;T&gt;</code></a> that implements<br>
    <a href="https://github.com/dotnet/orleans/blob/master/src/Orleans/Streams/Core/IAsyncObserver.cs"><code class="highlighter-rouge">Orleans.Streams.IAsyncObserver&lt;T&gt;</code></a> and
    <a href="https://github.com/dotnet/orleans/blob/master/src/Orleans/Streams/Core/IAsyncObservable.cs"><code class="highlighter-rouge">Orleans.Streams.IAsyncObservable&lt;T&gt;</code></a> interfaces.
</p>

<p>In a typical example below a device generates some data, which is sent as an HTTP request to the service running in the Cloud. Orleans client running in the front end server receives this HTTP call and publishes the data into a matching device stream:</p>

<div class="language-csharp highlighter-rouge">
    <pre class="highlight"><code><span class="k">public</span> <span class="n">async</span> <span class="n">Task</span> <span class="nf">OnHttpCall</span><span class="p">(</span><span class="n">DeviceEvent</span> <span class="n">deviceEvent</span><span class="p">)</span>
<span class="p">{</span>
    <span class="c1">// Post data directly into device's stream.
</span>     <span class="n">IStreamProvider</span> <span class="n">streamProvider</span> <span class="p">=</span> <span class="n">GrainClient</span><span class="p">.</span><span class="nf">GetStreamProvider</span><span class="p">(</span><span class="s">"myStreamProvider"</span><span class="p">);</span>
    <span class="n">IAsyncStream</span><span class="p">&lt;</span><span class="n">DeviceEventData</span><span class="p">&gt;</span> <span class="n">deviceStream</span> <span class="p">=</span> <span class="n">streamProvider</span><span class="p">.</span><span class="n">GetStream</span><span class="p">&lt;</span><span class="n">DeviceEventData</span><span class="p">&gt;(</span><span class="n">deviceEvent</span><span class="p">.</span><span class="n">DeviceId</span><span class="p">);</span>
    <span class="n">await</span> <span class="n">chatStream</span><span class="p">.</span><span class="nf">OnNextAsync</span><span class="p">(</span><span class="n">deviceEvent</span><span class="p">.</span><span class="n">Data</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
</div>

<p>In another example below a chat user (implemented as Orleans Grain) joins a chat room, gets a handle to a stream of chat messages generated by all others users in this room and subscribes to it. Notice that the chat user neither does not need to know about the chat room grain itself (there might not be such a grain in our system) nor about other user in that group that produce messages. Needless to say, to produce to the chat stream, users don’t need to know who is currently subscribed to the stream. This demonstrates how chat users can be completely decoupled in time and space.</p>

<div class="language-csharp highlighter-rouge">
    <pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">ChatUser</span><span class="p">:</span> <span class="n">Grain</span>
<span class="p">{</span>
    <span class="k">public</span> <span class="n">async</span> <span class="n">Task</span> <span class="nf">JoinChat</span><span class="p">(</span><span class="kt">string</span> <span class="n">chatGroupName</span><span class="p">)</span>
    <span class="p">{</span>
    <span class="n">IStreamProvider</span> <span class="n">streamProvider</span> <span class="p">=</span> <span class="k">base</span><span class="p">.</span><span class="nf">GetStreamProvider</span><span class="p">(</span><span class="s">"myStreamProvider"</span><span class="p">);</span>
    <span class="n">IAsyncStream</span><span class="p">&lt;</span><span class="kt">string</span><span class="p">&gt;</span> <span class="n">chatStream</span> <span class="p">=</span> <span class="n">streamProvider</span><span class="p">.</span><span class="n">GetStream</span><span class="p">&lt;</span><span class="kt">string</span><span class="p">&gt;(</span><span class="n">chatGroupName</span><span class="p">);</span>
    <span class="n">await</span> <span class="n">chatStream</span><span class="p">.</span><span class="nf">SubscribeAsync</span><span class="p">((</span><span class="kt">string</span> <span class="n">chatEvent</span><span class="p">)</span> <span class="p">=&gt;</span> <span class="n">Console</span><span class="p">.</span><span class="n">Out</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="n">chatEvent</span><span class="p">));</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre>
</div>

<hr>

<h2 id="quick-start-sample">Quick Start Sample</h2>

<p>
    The <a href="Streams-Quick-Start">Quick Start Sample</a> is a good quick overview of the overall workflow of using streams in the application.
    After reading it you should read the <a href="Streams-Programming-APIs">Streams Programming APIs</a> to get a deeper understanding of the concepts.
</p>

<h2 id="streams-programming-apis">Streams Programming APIs</h2>

<p>A <a href="Streams-Programming-APIs">Streams Programming APIs</a> provides detailed description of the programming APIs.</p>

<h2 id="stream-providers">Stream Providers</h2>

<p>
    Streams can come via physical channels of various shapes and forms and can have different semantics. Orleans Streaming is designed to support this diversity via the concept of <strong>Stream Providers</strong>, which is an extensibility point in the system. Orleans currently has implementation of two stream providers: TCP based <strong>Simple Message Stream Provider</strong> and Azure Queue based <strong>Azure Queue Stream Provider</strong>.
    More details on Steam Providers can be found at <a href="Stream-Providers">Stream Providers</a>.
</p>

<h2 id="stream-semantics">Stream Semantics</h2>

<p>
    <strong>Stream Subsription Semantics</strong>:
    Orleans Streams guarantee Sequential Consistency for Stream Subsription operations. Specificaly, when consumer subscribes to a stream, once the <code class="highlighter-rouge">Task</code> representing the subsription operation was successfuly resolved, the consumer will see all events that were generated after it has subscribed. In addition, Rewindable streams allow to subscribe from an arbitrary point in time in the past by using <code class="highlighter-rouge">StreamSequenceToken</code> (more details can be found <a href="Stream-Providers">here</a>).
</p>

<p>
    <strong>Individual Stream Events Delivery Guarantees</strong>:
    Individual event delivery guarantees depend on individual stream providers. Some provide only best-effort at-most-once delivery (such as Simple Message Streams), while others provide at-least-once delivery (such as Azure Queue Streams). It is even possible to build a stream provider that will guarantee exactly-once delivery (we don’t have such a provider yet, but it is possible to build one with the <a href="Streams-Extensibility">extensability model</a>).
</p>

<p>
    <strong>Events Delivery Order</strong>:
    Event order also depends on a particular stream provider. In SMS streams, the producer explicitelly controls the order of events seen by the consumer by controlling the way it publishes them. Azure Queue streams do not guarantee FIFO order, since the underlaying Azure Queues do not guarantee order in failure cases. Applications can also control their own stream delivery ordering, by using <code class="highlighter-rouge">StreamSequenceToken</code>.
</p>

<h2 id="streams-implementation">Streams Implementation</h2>

<p>The <a href="Streams-Implementation">Orleans Streams Implementation</a> provides a high level overview of the internal implementation.</p>

<h2 id="streams-extensibility">Streams Extensibility</h2>

<p>The <a href="Streams-Extensibility">Orleans Streams Extensibility</a> describes how to extend streams with new functionality.</p>

<h2 id="code-samples">Code Samples</h2>

<p>More examples of how to use streaming APIs within a grain can be found <a href="https://github.com/dotnet/orleans/blob/master/test/TestGrains/SampleStreamingGrain.cs">here</a>. We plan to create more samples in the future.</p>

<h2 id="more-material">More Material</h2>

<ul>
    <li><a href="https://plus.google.com/events/crdjm977pqubv81lgdhqruoqal4">Orleans Virtual Meetup about Streams</a></li>
    <li><a href="http://dotnet.github.io/orleans/Presentations/Orleans Streaming - Virtual meetup - 5-22-2015.pptx">Orleans Streaming Presentation from Virtual Meetup</a></li>
</ul> 